An Evaluation To Detect And Correct Erroneous Characters Wrongly Substituted, Deleted And Inserted In Japanese And English Sentences Using Markov Models

K e y words: Markov model, error detection, error correction, bunsetsu, substitution, deletion, insertion 1 I n t r o d u c t i o n In order to improve the man-machine interface with computers, the <tevelopment of input devices such as optical cha.racter tea<lets (OCR) or speech recognition devices are expected, llowew;r, it is not easy to input Japanese sentences J)y these devices, because. they are written by many kinds of characters, especially thousands of "kanji" characters. The sentences input through an OCR. or a speech recognition device usuMly contain erroneous character strings. The techniques of natural language processing are expected to find and correct these errors. tIowever, since current technologies of natural language analysis have been developed for correct sentences, they cannot directly be applied to these problems. Up to now, statistical approaches have been made to this problem. Markov mo<lels are considered to be one of" machine learning models, sinfilar to neural networks a.nd fuzzy models. They have been applied to character chains of natural lang,,a~ges (e.g.,l);nglish)[l],[2], a.nd to phoneme reco~gnition 3 . [41 cha.ins in continuous speech. . [ 1~1. ¢1' 2nd-orde.r Markov model nt bunsets',l is known to be useful to correct errors in "kanjikana." "/m nsetsu" [(;],to choose a correct syllable chain from Japa.nese syllable "bunsetsu" candidates [7], and to re(!nce the ambiguities in translation processing of non-segmented "kana." sentences into "kanji-kana" sentences [8]. The erroneous characters can be classilied Ul,O three types, lhe hrst is w~ongly recognized chal;aclers instead of correct (haracters. The second and the third are wrongly inserted and deleted (skipped) characters respectively. Markov chain mode.Is above mentioned were restricted to tind and correct the first type of errors[5],[6]. No method has been proposed for correcting errors of the second and the. third types. 'Phe. rea.son might be considered to be I.he di[ticulties of finding the error location and distinguishing between deletion and insertion